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Developing Institutional Indicators: The Role of Institutional 
Research 



Abstract 

Universities are coming under increasing pressure to demonstrate accountability in operations 
that affect student enrollment and that contribute to the increased cost of higher education. 
Institutional researchers are responding by working to provide strategic data-driven decision 
support that enables managers to evaluate the benefit of dollars spent on both instructional 
activities and non-classroom activities. While tools such as key performance indicators are 
useful for study of traditional activities, these tools frequently lack the flexibility to describe and 
generate all types of data required by the diverse, complex non-classroom activities of successful 
universities. This paper demonstrates how this problem can be addressed by involving relevant 
personnel in identifying mission-based success factors, indicators and learning assessments 
within key decision domains. A methodology is demonstrated that links assessed outcomes in 
Student Affairs to University strategic purposes. 



What We Do 

“Information anxiety is produced by the ever widening gap between what we understand and what we think we 
should understand. It is the black hole between data and knowledge, and it happens when information doesn't tell us 
what we want or need to know.” (Wurman, R. S. 1989. Information anxiety. New York: Doubleday.) 
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Introduction 



Universities are coming under increasing pressure to demonstrate accountability in 
operations that affect student enrollment and that contribute to the cost of higher education. 
Institutional researchers are responding by working to provide strategic data-driven decision 
support that enables managers to evaluate the benefit of dollars spent on both instructional 
activities and non-classroom activities. While tools such as key performance indicators can be 
shown to be useful for study of traditional activities, these tools frequently lack the flexibility to 
describe and generate all types of data required by the diverse, complex non-classroom activities 
of successful universities. For example, institutional managers need to answer questions 
concerning how expenditures for non-classroom, non-athletic student activities contribute to 
successful learning processes or to the overall success of academic program management. Data 
needed to answer such questions may not be found in all university databases. 

This paper demonstrates how this problem can be addressed by involving relevant 
personnel in identifying success factors and indicators within key decision domains. The 
conclusions show that our institutions are continuing the move toward a data-informed decision 
process to include accomplishments at multiple levels of the university. With the partnership of 
Institutional Research and managers in functions such as Student Affairs, our universities can 
move toward the strategic capabilities we need to be sustainable in the future. 

Approach and Methodology for Identifying Indicators 

It is generally accepted that becoming a strategically managed university involves the use 
of metrics in a manner that causes our key activities and functions to be consistent with the 
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mission of the institution. This requires that decision makers go beyond the singular 
identification of expenditures for instructional services to a description of other value-added 
activities within the context of the mission. Even in cases where institutions have no pressing 
public mandate to "operate efficiently and with accountability," regional accreditations agencies 
encourage managers to recognize the need is to develop a culture of evidence and awareness of 
analytics consistent with its core values. We have sought to raise the analytical awareness at our 
institution by engaging in projects that use traditional strategic management tools and 
techniques. Tools such as the balanced scorecard have evolved to meet the specific needs of the 
university. We have also developed underlying conceptual models for situations where data 
needs surface. The case used to demonstrate these projects is “The Student Affairs Assessment 
Initiative.” 

The Student Affairs Assessment Initiative 

Since January 2003, the Student Affairs Division has engaged the university community 
in a process to define and shape the “University Student Experience.” The goal is to enhance the 
quality of life and the learning environment for all students. This effort supports several of the 
University Learning Goals such as increasing the understanding of multiple cultures. The 
outcome of this effort has been the development of a long-range strategic plan that includes the 
goals and strategies that support and enhance a successful student experience. Consistent with 
good practices in higher education, the comprehensive strategic plan includes an assessment plan 
that (1) measures how the Student Affairs Division is meeting its stated goals and (2) strengthens 
the work of the division by building systems of accountability and continuous improvement. 
With the inclusion of an assessment plan, named the Student Affairs Assessment Initiative, the 
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project becomes part of the integrated university strategic plan that both enriches the lives of 
students and greatly contributes to the overall enactment of the university’s mission. 

Two key components comprise the Student Affairs Assessment Initiative — the Key 
Activities Report and the Learning Outcomes Assessment. The components are represented in 
Exhibit 1. The Key Activities component addresses the question: “What do you do?” Once 
completed, this report serves as a “snapshot” of the function’s activities, performance indicators 
and measurements. The second component of the Assessment Initiative, the “Learning Outcomes 
Assessment,” addresses the question, “So what?” In other words, Student Affairs assesses what 
they are doing and what students are learning from the programs and services they offer. 

Exhibit 1 




The action plan for implementing the Student Affairs Assessment Initiative laid out 
procedures and timelines for facilitating completion of the initiative. The tentative outline, due 
date, and suggested questions provided to jump-start the process are shown in Exhibits 2. In 
addition, the Student Affairs Division institutionalized its assessment activities through five 



5 








activities - (1) development of a Student Affairs Assessment Committee, (2) establishment of a 
connection with the Academic Affairs Office of Teaching, Learning and Assessment, (3) 
creation of the “Student Affairs Fact Book”, (4) involvement of Student Affairs in the North 
Central Association Review, and (5) development of a Student Affairs Annual Report. They are 
taking an active role in explaining to the state how the university is supporting and implementing 
the state performance indicators. 



Exhibit 2 



1. Departmental 2004-05 Student Experience Report - 

A. Mission 

B. Goals 

C. Strategies 

2. Key Activities Report - 

For each Key Activity: 

A. How would you measure: 

❖ Cost? 

♦♦♦ Magnitude? 

❖ Satisfaction? 

B. What are the learning outcomes? 

3. 2004-05 Learning Outcomes Assessment Report - 

A. Mission 

B. Goals 

C. Key Activities & Learning Outcomes 

D. Assessment Project(s) 2004-05 Academic Year 

E. Assessment Methods 

F. Implementation of Assessment Project(s) 

♦♦♦ Who is responsible for what? 

♦♦♦ Timeline 



The goals associated with the Student Affairs Initiative emerged out of its links to the 
university mission and were related both to support of the University’s efforts in quality 
improvement and its accreditation efforts. To meet these goals, Student Affairs personnel were 
engaged in discussions and workgroups to develop assessment at two levels. Level I was defined 
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at the level of the departments within the various areas of Student Affairs. These departments 

are where the programs and activities of Student Affairs are conducted. Level II was defined as 

the three areas of the Student Affairs Division - (1) Diversity Education/Leadership, (2) Student 

Advocacy/Community Relations, and (3) Student Development. This is the management level of 

Student Affairs. The following charges were communicated for each level: 

Level I-Each department within an area of Student Affairs was asked to identify 
(at least) one question about student learning, engagement, or interest in the 
department and document its answer each year; and 

Level II -The three areas of Student Affairs were asked to reflect on the individual 
department assessment reports within their area and to provide and document 
feedback to the units. The three areas could also engage in an assessment that 
cuts across the individual departments. 

Exhibit 3 describes the basic steps for the individual departments. The most meaningful 
questions about learning/ engagement/interest in departments are expected to come out of 
conversations already taking place in the departments. These questions in turn provide the 
foundation for identification of performance indicators that are relevant to assessment of these 
activities. As a partner in the process, Institutional Research has been working with the 
departments to locate areas of possible inquiry and strategies for assessment. These questions 
are guided by the goals of Student Affairs, the University mission, and the University learning 
goals. The breadth of some questions may necessitate that assessment be spread over several 
years. In such cases, the division is asked to examine a different aspect of the question or 
problem each year and thus, document progress in assessment for accreditation and program 
reviews. 
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Exhibit 3 



Level I - Each Department 


The assessment process for each department 

consists of three basic steps: 

• Posing and answering one question each year 
about some aspect of student learning, 
engagement, or interest: 

• Proposing any necessary changes to improve 
learning/engagement/interest; 

• Documenting this process. 


Report Questions for Level I - Departments 

1. What question was asked? What group of 
individuals did the question focus on? 

2. Describe how the question was answered. 

3. What was learned? 

4. What actions did or will the department take 
or consider to improve learning/ 
engagement/ interest? 

5. Are follow-up studies planned? 

6. What can Student Affairs and/or University 
do to help? 

7. What actions, if any, did you take based on 
last year’s assessment findings? 


Sample Assessment Questions for departments 

• To what extent do students participating in 
events/workshops value and use knowledge 
gained from the event/workshop? 

To answer the question, the departments might 
develop and administer a survey to students 
asking questions such as: To what extent did 
your knowledge or awareness of ‘the topic’ 
increase? What is the likelihood you will 
participate in upcoming similar events? What 
information from the event might you use and 
incorporate in your courses/work-life/social- 
life? 

• Which of our events drew the largest number 
of participants and why? How can we use this 
information to improve attendance in the 
future? 

• Which elements of our publication are most 
interesting/helpful and which are less 
interesting/helpful to our readership? 


Annual Calendar for 
Departments 

Nov. 1 -Submit proposed assessment question 
to TLA (tla@MdWest.edu) 

July 1 -Submit assessment report to TLA 
(tla @ MdWest.edu) 
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Exhibit 4 describes the basic steps at Level II for the three areas in the Student Affairs 
Division. The three areas of Student Affairs were asked to engage in assessment in two different 
ways: (1) by reflecting on department assessment reports to provide and document feedback to 
departments; and (2) by identifying one question about student learning in the area as a whole 
and documenting its answer each year. 



Exhibit 4 



Level II - Areas of Student Affairs: Diversity Education/Leadership, Student 
Advocacy/Community Relations, and Student Development 


I. Part I -Analyzing the department/program 
reports 

The success of the departmental assessment rests in 
large part on the departments' sense that their work 
will truly impact student learning and engagement. 
To this end. this paid of the Assessment Process 
asks that the three Student Affairs Areas review 
and reflect on department assessment reports — 
analyzing them, noting any proposed changes, and 
sharing the best practices in assessment that appear 
in the reports with all departments. We further ask 
that the Areas document the way that they provided 
feedback to departments, for example, by having 
the Student Affairs VP or A VPs write an individual 
letter to each department or by distributing the Area 
Report. In any case, we strongly suggest that the 
Area AVPs and VP become highly involved in the 
review and feedback process, as it is their 
recognition that matters most to departments. 


Part I Report Questions 

1. Summarize the department reports and/or 
assessments. 

2. Analyze and comment on the department 
reports. 

3. Summarize the actions that each department 
proposes to take to improve student learning. 

4. Describe the best assessment practices you see in 
the departments. 

5. Describe how you gave feedback to the 
department directors and staff. 

6. How can the University TLA Office support the 
assessment process in your Area? 


Part II - Assessing one aspect of student 
learning 

in the Area as a whole 

The second part of the assessment process for each 
Area consists of three basic steps: 

1 . Posing and answering one question each year 


Part II Report Questions 

■ What question was asked? 

■ Describe how the question was answered. 

■ What was learned? 
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about some aspect of student 
learning/engagement/interest, 

Proposing any necessary changes to improve 
student learning/engagement/interest; 

3. Documenting this process. 

We believe that the most meaningful questions 
about student learning/engagement/interest will 
come out of conversations already taking place in 
the Area. That said, given sufficient time, both 
TLA and OIPR are happy to help Student Affairs 
locate areas of possible inquiry and strategies for 
assessment. Some questions may be so large that 
you may wish to spread your assessment over 
several years, so long as you examine a different 
aspect of the question or problem each year and 
thus, document progress in assessment for NCA. 

In order to give OIPR the time to support your 
inquiry with any data that you request, we ask that 
you contact OIPR directly. Completed reports are 
due on the 15 th of November. 

Sample Area Assessment Projects: 

Study of factors that affect learning and retention 
o Student Success - general satisfaction with 
MdWest; career preparation; academic 
performance; satisfaction with career 
development 

o Academic Experiences - student reported 
information about, overall academic 
experience, campus-life experience 
o Engaged Learning - learning outside of 
class 

o Campus Environment - advising, 

inclusiveness, friendships, relationships, 
support, satisfaction ratings 
o Mission - respect for individuals, issues of 
diversity, integration of the Mission, 
affordability 



■ What actions did or will the Area take or 
consider? 

■ Are follow-up studies planned? 

■ What can Student Affairs and/or the 
University do to help? 

■ What actions, if any, did you take based on 
last year’s assessment findings? 



Annual Calendar for 
Student Affairs Areas 

July-Nov. Read department reports and 

conduct its own study of student 
learning/ engagement 

Nov. 15 Submit College Report to TLA 
( tla@MdWest.edu) 

Jan - Mar. Meet with TLA Office to discuss 
assessment 



The process described above is not isolated from broader endeavors at the University. In 
fact, it is based on conceptual and managerial models that are being institutionalized in the 
administration of higher education to help manage its unique abilities, opportunities, and 
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challenges. As noted, institutional indicators can be powerful tools in that effort, especially 
when development of the indicators incorporates attention to the contextual and operational goals 
of the university. Furthermore, they can provide substantive information for strategic decision 
making if they are tied to the values and goals of the university and emanate from the 
university’s critical success factors(Stewart and Carpenter-Hubin, 2001). The Student Affairs 
Division identified success factors and documented outcomes that act as data points in 
assessment of the function. These and the related concepts will be discussed in the following 
section. 



The Strategic Context of Higher Education - Laying the Foundation 
through Identification of Relevant Factors 

Critical and Strategic Success Factors 

Universities need to be successful in some functions in order to survive and complete 
with their peer institutions. They need to provide a safe environment in which they educate 
students, attract students consistent with their mission, and educate students for a cost consistent 
with the benefits and services provided. These are key activities for Student Affairs at which they 
must be successful and through which they successfully address critical and strategic success 
factors. 

Unfortunately, too frequently, the term “critical success factor” (CSF) is used 
interchangeably with the term “strategic success factor” (SSF). The difference is itself critical. 
While strategic success factors refer to those competencies at which the institution must succeed 
in order to be competitive or gain an advantage in the future, critical success factors are those 
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industry-defined competencies which all institutions competing in the industry must possess in 
order to survive and perform well. Failure to successfully perform on critical success factors 
leaves the institution vulnerable. Activities that support the critical success factors should thus 
be given priority. Discussions should differentiate between the two concepts as a means by 
which the Student Affairs Division’s activities and contributions to the University can be 
evaluation. 

In their classic work on critical success factors, MIT researchers John Rockard and 
Christine Bullen (1981) defined critical success factors as “the limited number of areas in which 
satisfactory results will ensure successful competitive performance.” In business terms, they are 
the few key areas where "things must go right" for the business to flourish and for the manager's 
goals to be attained.” (Bullen and Rockart, 1981: 7) A graphic representation is shown in the 
Figure 1 that is taken from the work by Bullen and Rockart. In higher education terms, critical 
success factors are the few areas in which “things must go right” for the institution to flourish 
and fulfill its mission. 

For Student Affairs, the concept of critical success factors would translate as the factors 
that must be addressed in order for the function and its areas to successfully support the mission 
of the University. The Student Affairs Assessment Initiative is one means by which the division 
can begin to better understand what those few critical factors might be. It should also be noted 
that the Bullen and Rockart definition of a CSF has several characteristics that are relevant to this 
effort by Student Affairs. First, the CSF concept is hierarchical in that for a factor to be evaluated 
as having satisfactory results, multiple initiatives must first be determined to have a successful 
outcome. Second, the factor is derived from what administrators within higher education agree is 
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important to success of Student Affair’s functions, even though the individual administrator or 
university may have a somewhat different perspective on the desirability of specific outcomes. 
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Figure 1 

Demonstration of Critical Success Factors 



FROM A CORPORATE VIEWPOINT 








As noted, strategic success factors differ in that they are those areas in which an 
institution must excel if it is to gain and maintain a competitive position into the future. This 
necessitates that an institution be compared with other institutions with which it competes for 
students, faculty, donations, etc. Meeting the standard for critical success factors may enable an 
institution to sustain itself in a relatively stable environment but not enable the same institution 
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to gain and maintain a competitive advantage into the future where there exists a more volatile 



environment. The challenge for the Student Affairs function is to understand the types of 
activities it must be successful at in order to survive in the present and which types it must 
implement in the present to prepare for the inevitable changes in the future. This need 
necessitates that Student Affairs functions pay close attention to both categories of factors, i.e., 
critical success factors and strategic success factors. It is useful to have a conceptual framework 
or methodology to discuss the components of activities that support success. The University has 
addressed this need by developing a framework called The Program Portfolio through which a 
balanced approach to identifying factors can be pursued. It is based on a model proposed by 
Kaplan and Norton in their Balanced Scorecard but is modified to be consistent with the 
guidelines of regional accreditation and the linguistic culture of colleges and universities. We 
used this framework in working with our Student Affairs Division to further refine their search 
for critical and strategic success factors. The following discusses the Balanced Scorecard 
concept. 



The Balanced Score Card 

Kaplan and Norton’s introduction of the Balanced Scorecard into the business literature 
gave notice to managers that a more balanced approach to evaluating an organization was 
needed. In other words, concentrating on financial well-being to the exclusion of customers, 
business processes, and learning and growth would lead to sub- optimization of outcomes. A 
more balanced approach, placing the mission of the organization as the center of concern, was 
needed. 
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Universities, such as Jacksonville University in Jacksonville Florida, have used the basic 
principles underlying the model to develop a set of perspectives that are useful for institutions. 
They include categories such as (1) Students, (2) Financial, (3) Internal Processes, and (4) 
Development and Growth to describe their university’s strategic objectives. A set of additional 
questions were identified to address department- and program- level areas: 

1. What are your objectives in this area (no more than four)? 

2. What are your strategic measures (for each objective, no more than three)? 

3. For each measure, what is your baseline? 

4. What are the next three targets with time points in which they will be 
sought? 

We also built on the balanced scorecard concept to create a Program Portfolio model. The goal 
of the model is to capture this same dynamic but adapting the concept to the unique aspects of 
higher education. (See http://oipr2.MdWest.edu/portfolio/MdWest.asp .) The model shown in 
Figure 2 was initially used to support Academic Program Review throughout the university as a 
means for developing questions to guide program assessment. This later became the basis for a 
current initiative in looking at the success factors for the university (Chmielewski, et al., 2001). 
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Figure 2 

The Program Portfolio 




The Program Portfolio has been used in several projects to organize university data to 
describe and then reflect of the key aspects of several of our departments. Used in this manner it 
has provided guiding questions to generate thee discussion of “Who are we?” “What do we do?” 
“How do we do it?” “What do we accomplish?” As we have entered these discussions, we have 
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evolved a set of indicators that apply to a broad range of university functions. By identifying 
these as factors at which we need to be successful,, we were able to facilitate a discussion with 
the Student Affairs Division in how to develop indicators. We thus used the balanced scorecard 
concept as a framework for discussing critical and success factors and indicators. This helped the 
Student Affairs Division identify and discuss the key activities they needed for their assessments. 

The Program Portfolio as it evolved for use by the university and bt Student Affairs 
Division is made up of five components — Learning Environment, Student Success, Program 
Vitality, University Sustainability, and Mission and Values. The model is shown in Figure 3. 
Each of the five components is described below: 

1. Learning Environment: Do the activities create a strong learning environment? This is 
the university’s primary function and is inherent in both its instructional goals for Vision 
2006 and in the activities of the Student Affairs Division. While it is generally 
recognized that Student Affairs can contribute to a better student environment through 
activities that support learning, service, and preparation for living in a diverse world, the 
charge to the Student Affairs Division is to determine how to measure those activities in 
order to establish accountability. Under the Student Affairs Assessment Initiative, 
questions can be generated that highlight what attributes of the environment to measure, 
how to measure the impact of the activity on the environment, how to identify which 
measures are related to factors that fit the definitions of critical or success factors, and 
what the implications for the future are given the desired and realized learning 
environments. 

2. Student Success: Do the activities support the development of students within the 
learning environment? If students are to be successful in the learning environment, there 
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needs to be a supportive learning and development environment that extends beyond the 
classroom and the academics of the university. This support includes helping the students 
select their academic experiences, develop the appropriate skills for a career, manage the 
financing of their education, and persist to graduation. Under the Student Affairs 
Assessment Initiative, the department can explore whether commonly used indicators 
such as graduation rates, student flow, job placement, and financial aid are also 
appropriate for use in assessment of the activities provided through the Student Affairs 
Division as a means to determine accountability. 

3 . Program Vitality: Are programs provided through the Student Affairs Division strong 
and viable? The activities and programs need to be sustainable in an economic sense. 
There needs to be an identifiable costbBenefit associated with the program. This includes 
having a core of relevant programs/activities, providing appropriate staffing those 
programs, and managing to improve the quality of the program as it supports the 
university and the student. It includes active student participation in the various 
programs. The challenge for the Student Affairs Division is to link measures of 
successful student outcomes to such things as program/activity participation, relevance, 
and contribution. Under the Student Affairs Assessment Initiative, the department can 
identify what types of data are available and what types of data must be collected in order 
to make the needed linkages between programs and outcomes. 

4 . University Sustainability: Are programs provided through the Student Affairs consistent 
with university efforts to remain healthy in terms of resources, alumni support, 
administrative compliance, attractiveness to potential students, debt burden, and 
financial stability and facilities well-being? If we are to continue to maintain 
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institutional relevance in our community and to continue to strengthen our educational 
competency, we must balance our commitments with our resources. Financial well-being 
is important to attracting and sustaining a quality faculty and staff and to maintaining 
programs and facilities. The support of alumni is closely tied to donations and 
development. The challenge of Student Affairs is to demonstrate how activities contribute 
to the bottom-line sustainability of the university. Under the Student Affairs Assessment 
Initiative, such traditional measures as retention and participation rates in student affairs 
activities can be examined for possible relevance to financial health. In addition, the 
department can use the initiative as a means for identifying other data that can be 
gathered for examination of performance on critical and strategic success factors related 
to finance and facilities. 

5. Mission and Values: Are activities consistent with the mission and core values of the 
university? The fifth component, the University Mission and Values, is the center of our 
framework. This is symbolic of the fact that the mission and values describe the reason 
for MdWest’s existence. As such, all areas must operate in a manner consistent with the 
beliefs, purpose, tradition and values of MdWest. In other words, creating outcomes that 
support the mission of MdWest is the core around which other concerns revolve. This 
requires that contributions to all aspects of the university and its community must fulfill 
our mission, from education and scholarly activities to enhanced learning beyond the 
classroom such as that provided by the Student Affairs Division. 
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Figure 3: The Revised Program Portfolio 
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At the university level, Institutional Research linked a set of categories with each 
component in the Program Portfolio. For example, categories of concern under Sustainability 
include affordability, attractiveness, revenue activity, facilities utilization, investment 
performance, programs and partnerships, and risk management; categories of concern under the 
Learning Environment include active intellectual engagement, rigor, service learning, and 
diversity. For each set of categories, we identified four or five measures that could be used. For 
example, measures under the category “Active Intellectual Engagement” include sizes of classes, 
participation in clubs and organizations, interaction with faculty, and internships and capstone 
classes. Some of these measures currently exist and some of them do not. We use this as a 
general reference guide to discuss metrics within a framework of critical and success factors, 
some of which are currently available and others that are not yet available. In cases where 
measures have not been identified, such as Student Affairs, an iterative process by individuals 
working in the area supports the identification and creation of the appropriate measures. This is 
especially critical in instances where questions concerning understanding “why” certain activities 
lead to successful or unsuccessful outcomes need to be answered if good decision making is to 
be supported. 

Though still evolving, one of the desired outcomes from developing a program portfolio 
model is the identification of a set of Academic Performance Indicators at the university level to 
examine outcomes. (For example, see Sapp, M. 1994; MdWest, 2005 

http://oipr.MdWest.edu/open/Academic%20Indicators/indicators.asp .) These indicators are made 
available to our stakeholder and communicated effectively by grouping the indicators into the 
areas of Student Learning, Student Enrollment, Faculty, and Tuition and Financial Aid. The 
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same strategy is part of the process we using as we partnering with Student Affairs to generate 
development of sets of indicators at other levels of the university that support good decision 
making. 

We do want to stress, however, that while program portfolios, performance indicators, 
strategic models and use of critical or strategic success factors can be useful tools for supporting 
strategic decision making, choice of the best strategy requires judgment on the part of the 
decision maker. The role of these tools is to provide a template for considering the major aspects 
of key activities and for suggesting how the measures might be developed. The act of making 
judgments then needs to be made through reference to absolute or relative standards that are 
identified through use of these tools. For example of absolute standards, one might say that any 
student who enters their university should have at least a 75% likelihood of graduating. More 
frequently such judgments are made relative to some frame of reference for the data. The data 
may be presented in one of three major contexts - trends over time, external comparison, and 
internal comparison. Questions that surface when looking at trends over time concern whether 
the outcome is changing as time passes. Under the Student Affairs Assessment Initiative, 
individuals can assess what trends should and can be measured longitudinally. In addition, the 
nature of the change can be factored in. For example, can the Student Affairs Division determine 
whether an outcome is actually changing in a systematic manner over time or in a statistically 
stable way with only random variations? 

When overall performance of the university is compared to the overall performance of 
other universities, outcomes such as the retention rate at your university compared to similar 
universities can be evaluated. In the case where there are not sufficient similar universities for 
comparison, we can attempt to build a statistical similarity using various statistical modeling 
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techniques, e.g., multiple regression. At the university level, public domain data can be used 
where available so that assessments of competitors can be made with reasonable accuracy. 
Unfortunately, institutional researchers may not be able to drill down to the level of the Student 
Affairs Division for comparisons with similar institutions. Recognition of this problem through 
the Student Affairs Assessment Initiative can lay the groundwork for cooperation across and 
between colleges that can generate the needed data. 

The third type of judgment can be made using an internal comparison between departments of a 
College or areas within a division. For example, the Department of English may assess how it 
compares to the Department of Accounting on developing career skills. In the case of the 
Student Affairs Division, comparisons could be made between the three areas to consider the 
effectiveness of programs offered. For example where several areas have a similar goal of 
supporting student retention, the effectiveness of alternative student success programs might be 
considered. Other issues such as equitable financing, staffing balances, etc will be very difficult 
to do because of the differences in the purposes and processes of the areas. When indicators are 
considered in this manner relative to the purpose of the area, a key vulnerability might be 
identified where the area lacks the required skills or resources to function or compete effectively. 
On the other hand, where outcomes are favorable, the department can identify the core 
competencies on which future activities can be built. Core competencies are thus the key to 
department’s ability to develop distinctive competencies that support the university in its efforts 
to build a competitive advantage. The use of absolute or relative standards enables the Student 
Affairs Division to set goals for their indicators. It also enables the Division to use the 
indicators to determine if a key activity is making a positive difference in some aspect of student 
learning. 
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The Supporting Role of Institutional Research and Performance 
Indicators 



Given their access to data and training that creates an awareness of what it means to be 
“data driven,” institutional researchers have an opportunity to provide assistance to various 
departments within the university by helping them link functional-level data to strategic decision 
making. Institutional researchers have a heightened appreciation of the problems associated 
with data management in general. They understand that the presence of data may not be 
sufficient if that data are not the “right data” in the “right form” at the “right time.” Their 
involvement in identifying performance indicators that can be used by universities to monitor 
outcomes gives them an advantage in understanding the processes involved in identifying critical 
and success factors. The challenge is to get the numerous divisions and departments throughout 
the university (1) to buy-in to the need for data-driven processes and (2) to use the resources 
made available to them by their Office of Institutional Research. 

The catalysts for appreciation of data-driven processes and the need for support services 
generally take two forms — 1) the daunting process of preparing for accreditation reviews or 2) 
involvement in the strategic management process. For many universities, the latter process is 
being institutionalized. MdWest is currently refining and redefining its vision as it moves from 
its current strategic planning process, Vision 2006, to its next planning process, Vision 2012. 

The models and tools presented were developed to support the implementation of plans and 
assessment strategies associated with this initiative. MdWest University, like other universities, 
works to avoid the pitfall noted by Stokes (2002) who warns that that strategic planning 
exercises often fail because plans are frequently not implemented and strategy is forgotten. We 
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recognize the need to create support for strategic planning through a balanced approach. This 
approach must be sensitive to all aspects of planning, including growth and innovation with cost 
and efficiency, short term and long term objectives, internal and external views, and tangible and 
intangible results (Niven, 2002). 

It is thus important to link the critical and strategic success factors to specific problems to 
be solved, decisions to be made, and resources to be allocated. Furthermore, the strategic 
management process must link unit goals to macro goals while focusing on both the tangible and 
intangible. Through this process, the Office of Planning and Institutional Research strives to 
support the university's efforts to develop strategies that lead to successful outcomes on success 
factors at all levels of the university. The performance indicators identified through the Student 
Affairs Assessment Initiative contributes to this effort and can be used to evaluate outcomes and 
to revise strategies as appropriate. 

Student Affairs professionals have also recognized the need to address and identify 
success factors. For example, there are an increasing number of instruments and sites for 
Student Personnel professionals who want to assess specific aspects of the student learning . 
Among them are the sites developed by ACPA, the leading student affairs Association concerned 
with engaging students in lifetime learning and discovery. (See 

http://www.myacpa.org/au index.cfm .) The association’s Commission on Assessment for 

Student Development Clearinghouse on Environmental and Student Development Assessment 

Instruments lists between 50 and 100 instruments organized along the subjects such as: 

Career Related Instruments 
Environmental Assessment Tools 
Higher Education Evaluation Tools 
Learning Styles Assessments 
Miscellaneous Assessment Instruments 
Outcomes Assessments 
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User Friendly Personality Instruments 
Retention Measures 
Student Development Instruments 
Values Measures 

The ACPA initiatives exist within a broader initiative of assessment of learning - of 
which there are some excellent web sites that link together many of the current activities and 
issues. These sites, such as http://www2.acs.ncsu.edu/UPA/assmt/resource.htm at North 
Carolina State, make the process of measuring the learning a much more informed process. 

Other sites, such as http ://www2 .poly .edu/Info/Links 1 . shtml of the Polytechnic 
University of Brooklyn, provide listings of many of the assessment web sites at colleges and 
universities with a short description of what is on the sites. Unfortunately only two of the 
mentioned institutions consider Student Affairs as part of their description. 

On the other hand, the work done by Peter Ewell and others that evolved into the NSSE 
and the works that emerged from Ernest Pascarella and Patrick Terenzini. clearly indicate that 
the learning outside the classroom is a complementary aspect of student knowledge to that 
learned in the classroom in many cases. In other cases, the experiences outside the classroom 
may be the primary source of learning (NSSE, 2003). There are encouraging signs. For 
example, institutions such as the University of Hawaii at Hilo have moved to incorporate 
performance indicators from student affairs into their university goals. 
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Exhibit 5: University of Hawaii at Hilo Statement of Goal III 

Goal III: Build a learning environment that facilitates student development and 
success. 

We will design our services so that all our students - residential, commuting, and 
distance learners - may take maximum advantage of a learning environment truly 
conducive to educational effectiveness. 



Objective 


Indicator 


Data 

Gathering/Reporting 

Means 


Current Status 


Stimulating, 
supportive campus 
atmosphere 


Track the # of 
cultural, social, and 
athletic events on 
campus 


UH Office of Student 

Affairs/University 

Relations 




Stimulating, 
supportive campus 
atmosphere 


Track responses to 
questions in 
campus surveys 
about quality of 
campus life 


Graduating Senior 
Survey; CSEQ or in- 
house survey 


New question on 
the GSS asks 
students to rate 
quality of campus 
life and availability 
of things to do 


Increase capacity 
to serve 
commuting, 
nontraditional, 
distance learning 
students 


Assess special 
needs of 
commuting, 
nontraditional, 
distance students 


Extra questions in the 
CSEQ or in-house 
survey 


Awaiting results of 
first CSEQ 



http ://w w w . uhh.hawaii.edu/uhh/strategic/performance.php 



Progress in recognizing the need to move assessment beyond the classroom can also be 
traced to work by the American Association of Higher Education. For example, the 6 th principal 
of the American Association for Higher Education: Nine Principles of Good Practice for 
Assessing Student Learning in their recognition that “(s)tudent learning is a campus-wide 
responsibility, and assessment is a way of enacting that responsibility." The exact wording of the 
principle is as follows: 
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6. Assessment fosters wider improvement when representatives from across the 
educational community are involved. Student learning is a campus-wide responsibility, 
and assessment is a way of enacting that responsibility. Thus, while assessment efforts 
may start small, the aim over time is to involve people from across the educational 
community. Faculty play an especially important role, but assessment's questions can't be 
fully addressed without participation by student-affairs educators, librarians, 
administrators, and students. Assessment may also involve individuals from beyond the 
campus (alumni/ae, trustees, employers) whose experience can enrich the sense of 
appropriate aims and standards for learning. Thus understood, assessment is not a task for 
small groups of experts but a collaborative activity; its aim is wider, better-informed 
attention to student learning by all parties with a stake in its improvement. 
(http://www.aahe.org/assessment/principl.htm ) . 

In the final analysis, the strategic management process becomes a means for moving the 
university from external concerns focused on image and ranking to internally-driven concerns 
focused on improved institutional effectiveness. The Student Affairs Assessment Initiative is 
just one means for moving closer to that goal. It is one piece of a larger puzzle that includes 
other initiatives such as faculty committees involved in Teaching Learning and Assessment , 
the work being done on budgeting through Strategic Resource Allocation Committees, and the 
strategic planning done during a President’s Planning Retreat. 

Institutional Research can contribute by proposing conceptual models that demonstrate 
how the various factors discussed above can fit together. “The times they are a-changing.” 
Evaluating only student learning in the classroom is no longer sufficient. Assessment performed 
absent a sense of the larger university is no longer sufficient. A more holistic approach is 
needed. As institutions become more complex and as they move toward accreditation, there will 
be an increasing interest in identification of the tools available from a variety of sources. 
Technology has made use of many new and traditional tools easier, but the challenges we face 
are not technical challenges. They are challenges within the processes of higher education, its 
unique abilities, its unique opportunities, and its unique challenges. 
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